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The mathematical knowledge for teaching (MKT) measures have become widely used among researchers 
both within and outside the U.S. Despite the apparent success, the MKT measures and underlying 
framework have been subject to criticism. The multiple-choice format of the items has been criticized, and 
some critics have suggested that opening up the items might be an option. One way of opening up the items 
is to include commentary boxes that allow teachers to explain their thinking. This paper reports on a 
Norwegian study where commentary boxes were added to MKT items in order to investigate the 
connection between teachers’ responses to the items and their written reflections. The results indicate that 
there is a mismatch between the answers given by the teachers on the MKT items and their written 
reflections. Teachers’ written reflections do not always support their responses to the MKT items. 
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Introduction 


Knowledge about mathematical topics and teaching tasks with which teachers struggle is useful when 
preparing professional development (PD) programs (Hill, 2010). Various methods have been used to study 
and assess different aspects of teachers’ knowledge (e.g., Hill, Sleep, Lewis, & Ball, 2007). The work by 
Ball and colleagues at the University of Michigan (e.g., Ball, Thames, & Phelps, 2008) is well known, and 
they have developed the concept of “mathematical knowledge for teaching” (MKT). MKT is defined as 
“the mathematical knowledge used to carry out the work of teaching mathematics” (Hill, Rowan, & Ball, 
2005, p. 373). They have also developed sets of multiple-choice (MC) items to measure MKT. These 
MKT measures were designed from studies of the work of teaching mathematics in the U.S. (e.g., Hill, 
2010). The results from these researchers’ efforts are encouraging. MKT appears to make a difference to 
the mathematical quality of instruction (Hill et al., 2008), as well as to students’ achievements in 
mathematics (Hill, et al., 2005). Morris et al. (2009, p. 492) have described MKT as: “the most promising 
current answer to the longstanding question of what kind of content knowledge is needed to teach 
mathematics well.” 

Many researchers have built upon the efforts of Ball and colleagues, and the MKT measures have been 
widely used both within and outside the U.S. (e.g., Ng, 2012). Despite the apparent success of this 
research, there have also been critics (e.g., Schoenfeld, 2007). It is suggested, for example, that the 
knowledge required for teaching may be more culturally based than simply pertaining to mathematical 
knowledge (Stiegler & Hiebert, 1999; Stylianides & Delaney, 2011), and that cultural aspects have not 
been taken into consideration in the development and application of the MKT measures. There have been 
efforts to study the challenges of translating and adapting the items into a different cultural context (e.g., 
Fauskanger, Jakobsen, Mosvold, & Bjuland, in press; Ng, 2012) and to compare some of these challenges 
(Ng, Mosvold, & Fauskanger, 2012). Substantial additional investigation is needed to learn more about the 
cultural issues related to the translation, adaptation and use of MKT items in different cultural contexts. 
Another criticism relates to the MC format of the items. Schoenfeld (2007) claimed that the MC format has 
the potential to complicate the items for test-takers. This claim was supported by findings from a 
Norwegian study (Fauskanger, Mosvold, Bjuland, & Jakobsen, 2011). In the Norwegian study, teachers 
suggested that the items include commentary boxes to enable the teachers to explain their thinking, which 
has been proposed as one way to open up the items. Thus, an extended discussion of the validity of the 
items appears to be necessary. 

The present paper contains a discussion of the criticism as addressed by adding commentary boxes to 
enable the inclusion of teachers’ written reflections to the MKT items. One expectation would be that 
teachers responding correctly to an item would also provide reflections that support their responses, but to 
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our knowledge no previous attempts have been made to investigate opening up of MKT items in this way. 
We address herein the following research question: 


What is the connection between teachers’ responses to MKT items and their written reflections 
concerning the content of the items? 


Given the importance of developing students’ fluency in multi-digit arithmetic as a foundation on 
which to build a proper understanding of the decimal number system (e.g., Verschaffel, Greer, & 
De Corte, 2007), we have chosen to analyze data from a MKT testlet, including four items where different 
methods of decomposing a three-digit number are presented. 


Methods 


The research reported in this paper is part of a larger project focusing on teachers’ MKT and their 
corresponding beliefs about MKT (e.g., Fauskanger, 2012). For the purpose of this paper, data from 30 
teachers’ responses to MKT items and their written reflections will be analyzed, taken from one testlet 
including four MC items. This testlet has not been released for publication,' therefore we are only able to 
provide a description of it. The stem presents a context dealing with groups of students who have 
decomposed a three-digit number (e.g., 456) into hundreds, tens, ones and tenths in different ways. The 
question posed is which answer the teacher should accept as correct. In the first item (1a), the students 
have answered incorrectly (e.g., 456 decomposed into 4 hundreds, 50 tens and 6 ones). The remaining 
three items represent correct decompositions including hundreds, tens and ones (1b), hundreds, tens and 
tenths (1c) and tens and ones (1d). The decomposition that strictly follows the positions (e.g., 456 divided 
into 4 hundreds, 5 tens and 6 ones) is not present in any of the items. 

Although MKT items are normally used in a testing situation, other alternative uses have been applied. 
In our study, the teachers responded to the items at home. It is important to consider the advantages as well 
as disadvantages of allowing teachers to work with the MKT items at home (see e.g., Hill, 2011). One 
obvious consequence is that the teachers in our study had the opportunity to discuss the items with others. 
Their written reflections, however, were individual. They were asked to reflect on the following questions 
in the commentary boxes: (1) What do the students responding as in items (a) to (d) know? (2) What, if 
anything, do they need to learn more about? (3) Do the items in this testlet reflect a content that is relevant 
for the grade(s) you teach? (Why?/Why not? Please provide an illustrating example from your classroom). 
The reflections were provided for the entire testlet, not for each individual item. 

The 30 teachers (8 men and 22 women) were participating in a one and one-half year PD course, and 
the written work reported on was given as an assignment after their first day in this PD. Sixteen of these 
teachers worked in grades 1-4, nine in grades 5—7 and five in grades 8-10. Their working experience as 
teachers varied from less than 5 years to more than 20 years, and their formal education in mathematics 
(education) varied from 0 to 60 ECTS.” 

The analysis was conducted with the aid of the computer software NVivo9 (QSR International). The 
teachers’ written reflections were first divided into two groups. One group contained reflections from 
teachers who had identified the correct key for all four items in the testlet (main group 1, table 1), whereas 
the second group contained reflections from teachers who had responded incorrectly to one or more of the 
items (main group 2, Table 1). In the next phase of the analysis, a grounded theory approach (Strauss & 
Corbin, 1998) was applied in order to analyze the reflections from these two groups of teachers according 
to how they had commented on the items. Based on several cycles of reading and re-reading the data, the 
teachers’ reflections were refined into codes that were revised several times to establish consistency. The 
codes were based on well-established findings from the literature concerning place value (e.g., Jones, 
Thornton, Putt, et al., 1996; Verschaffel, Greer, & DeCorte, 2007). Two sub-categories (a and b in table 1) 
were discovered for each of the two main groups as a result of this analysis. 
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Results 


In this section, we present the results from our analysis of data regarding the connection between 
teachers’ responses to the four MKT items and their written reflections on the content of the items. Item la 
presents a context where a group of students had given an incorrect response to the MKT item. All 30 
teachers in our study identified the key in this item, but 10 teachers had incorrect responses to at least one 
of the other items in the testlet. If the teachers selected the alternative “I am not sure,” their answer would 
then be coded as incorrect (see Table 1). 


Table 1: Teachers’ Reflections Regarding Multiple Decompositions 


Think multiple decompositions are Think multiple decompositions are 
correct (1) incorrect (2) 


At least one incorrect 1 
(b) 


It is important to notice that it was not always evident in which groups the teachers should be placed; 
when placing the teachers in groups we allowed them the benefit of doubt if their reflections were 
ambiguous (e.g. Gerd’s reflection below). Two among the seven teachers in the first group could have 
been placed in the second group, but were placed in the first group although some thoughts in their written 
reflections were incomplete. 


Correct Responses with Supporting Reflections (Group la, Table 1) 


Although 20 teachers identified all four correct keys in this testlet, only seven displayed supporting 
thinking in their written reflections. Oda’ was one of the seven teachers who gave the correct answer to all 
four items, and she wrote this in her reflections on item Ic: 


I think we have before us an advanced solution in relation to the place value system (in this item). This 
student has a well-developed number concept and is able to use his fantasy when replacing the one 
with tenths. In this way, his knowledge about tenths is displayed. 


Two of the seven teachers indicate that the students display a very good understanding when they make a 
decomposition of numbers that differs from the standard decomposition. Tor writes: 


They could have given a more simple solution by using 4 hundreds, 5 tens and 6 ones, but we can say 
that some (students) are clever in the way they don’t necessarily use the correct decomposition but still 
get the right answer. 


The reflections of some teachers were less clear, and as a result it was difficult to evaluate whether or not 
they displayed a correct manner of thinking. In relation to item 1b, Gerd writes: 


This is, consciously or unconsciously, written in a more advanced way. It might be that he wants to 
show that he has complete mastery of this, or it might just be a coincident. 


These reflections indicate that Gerd believes that the students might have a more advanced understanding, 
but she is not certain about whether or not their responses are conscious. Later in her reflections, however, 
she writes that: “none of these solutions are perfect according to the place value system.” This last 
assertion is not explained further. 
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Correct Responses with Non-Supporting Reflections (Group 2a, Table 1) 


Of the 20 teachers who had correct responses to all four MC items in this testlet, 13 provided 
reflections that did not entirely support their responses. Dina writes: 


The students need to learn about the standard place value system and the proper exchange between 
digits. 


Brit writes: 


The students need to learn more about exchange, learn to fill up the ones, tens, hundreds, etc. Know 
that each position has its (distinct) value. When the value exceeds 9, they should shift position. 


In relation to item 1d, Frida writes that the students: 


... lack an understanding of the place value system, and the student only understands the tens place and 
ones place in the place value system. The total sum is still 456, so the student obviously understands 
decomposition and the value of the number (...) All students are on their way towards an understanding 
of how a number can be written in extended form. They have to learn more about the place value 
system. They need to reach an understanding of which number belongs where, one number in each 
position. 


The suggested solutions in items (b) through (d) are all mathematically correct, and these teachers have 
identified the correct solutions in the MC items. In their reflections, however, they seem to insist on the 
mathematical convention: “When the value exceeds 9, they should shift position.” Although they 
recognize that the students’ solutions were mathematically correct, they do not regard them as “the answer 
the teacher is seeking,” and, therefore, their reflections do not support their responses to the MC items. The 
written reflections of these teachers are in line with those given by the nine teachers in the last group (2b in 
Table 1), which consists of teachers who have identified one or more incorrect keys and who have written 
reflections supporting their responses to the MC items. For example, Erna identified three correct keys and 
writes this in her reflections: 


Student b) is wrong, but still right. Incorrect decomposition, but the correct total (amount). The student 
has understood how to decompose the number so that it doesn’t increase or decrease in value, but still 
hasn’t placed it correctly according to the place value system. 


Eli identified the correct key for item la only, and she writes this in connection with items 1b-d: 


b) This student manages to decompose the number 456, but apparently hasn’t completely understood 
the value of the digits in the place value system. It is indeed correct that you can decompose 456 into 3 
hundreds, 15 tens and 6 ones, but this is not the answer the teacher is seeking. 


c) This student has understood the value of the numbers 4 and 5, but is mixing up the ones. It is a little 
bit funny to see that the student makes it so hard on himself. This student knows that there are 10 
tenths in 1 one. 


d) This student knows how many tens there are in 456, but hasn’t understood the value of the digits. 


Which of these answers you should approve as correct depends on how long the students have been 
working on this topic. If the students have been working with this for an extended period of time, I 
wouldn’t have approved any of the answers. If, however, this is the introduction to decomposition of 
numbers, I would have approved b) through d). 


In their reflections, the teachers in the fourth group appear to insist on the same mathematical convention 
as the teachers in the second group do. 


Incorrect Responses but “‘Correct” Reflections (Group 1b, Table 1) 


Out of the ten teachers who gave incorrect responses to one, two or three items, or who gave the 
response “I am not sure” to some of these items, one teacher showed an understanding of the MKT being 
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measured in her reflections. Laura marked all three items with “I am not sure” (which is coded as an 
incorrect response), but she argued in her reflections that the testlet stem could be interpreted in different 
ways and that the key for each item would be dependent on how the stem was understood. The following is 
an excerpt from Laura’s reflections: 


Item a) is wrong by all means. Items b), c) and d) are wrong if it (the problem presented in the stem) is 
a closed problem, but they are correct if it is an open problem. 


By “closed problem” this teacher means using the positions given (e.g., 456 = 4 hundreds, 5 tens and 6 
ones) and by “open problem” the teacher means open to other ways of decomposing three-digit numbers. 
This teacher’s written reflections are in line with some of those from the seven teachers in the first group. 


Discussion and Conclusions 


Four groups (as presented in Table 1) emerged in our analysis, and the results from our study indicate 
that there is not always a clear connection between the teachers’ responses to the MKT items and their 
written reflections. Researchers who use the MKT items would probably expect, or at least hope, that 
teachers who answer the MC items in the measures correctly also have an appropriate understanding of the 
content, and the other way around. In our study, there are some teachers who follow this pattern. We have, 
however, identified an apparent mismatch between the responses to the MC items and the written 
reflections of several teachers. We have seen an example of one teacher (group 1b) who provided incorrect 
responses to the MC items but who displayed a high level of understanding in her written reflections. She 
appears to know the mathematics but she is still unable to determine for what answer the test-makers are 
looking. Another group of teachers (group 2a), were interesting as they were able to select the correct 
answer, but appeared to still believe that there is a particular name for the number that is better. These 
teachers are able to see that “4 hundreds, 5 tens and 6 ones” is the same value as “4 hundreds, 15 tens and 
6 ones,” but still believe that the name that matches the place values is better. If we consider the example 
of using the standard Norwegian algorithm for calculating 456 minus 37 (Figure 1), we have an example 
where the “place value name” is clearly not the best name for the number. As a result, teachers who hold 
such beliefs could be seen as in transition along a continuum. First, we have the teachers who are not able 
to understand non-standard decompositions of numbers. Second, there are teachers who can understand 
multiple decompositions of numbers, but who still believe that the standard decomposition is best. Third, 
there are teachers who understand and value multiple decompositions of numbers. Fourth and finally, there 
is a possibility that some teachers understand, value and can explain the use of alternate decompositions. 


10 
456 
me Tf 
419 


Figure 1: Standard Norwegian algorithm for subtracting 37 from 456 


The findings may be explained in a variety of ways. We present four possible explanations as follows. 


1. The findings are incidental. This might be connected with our study being limited by the 
number of participants as well as the limited focus of the items. More research is necessary in 
order to investigate whether or not the same tendencies can be found in a larger number of 
participating teachers. 

2. This apparent mismatch is specific to this particular topic. It would be pertinent to also 
investigate whether or not, or if the same pattern can be found for all sets of MKT items. 
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3. There are cultural differences involved in how teachers reflect upon these items. One such 
difference might be related to how the decimal number system is taught. If such cultural 
differences are involved, it would be of great interest to conduct additional research to 
investigate this further. Researchers have already adapted and used MKT items in different 
countries (e.g., Ng, 2012), and if there are cultural differences related to the connection 
between teachers’ responses to items and their reflections, great care should be taken when it 
comes to how the results from such studies are interpreted. We suggest that efforts should be 
made to investigate these issues both inside and across cultures to learn more about the 
connection between teachers’ MKT, their corresponding beliefs and the educational culture(s). 

4. A final possible explanation of the differences between these Norwegian teachers’ responses 
to the MKT items and their written reflections is that there are indeed, as Schoenfeld (2007) 
argued, constraints resulting from the MC format of the items. Such possible difficulties with 
this format might be specific to culture, as Fauskanger and colleagues (2011) suggest. From 
the results of the present study, however, it does not appear that the MC format itself makes it 
more difficult for the teachers. The complicating connection between the teachers’ responses 
to the MKT items and their written reflections only indicates that the MC items are harder to 
interpret than they might appear. The inclusion of commentary boxes along with the items, or 
other ways of opening up the items, should be investigated further. It would also be relevant to 
include interviews with teachers to further investigate teachers’ reflections as well as the 
connection between these reflections and their MKT as measured by their responses to the 
items. 


Our study indicates that researchers have to be careful concerning the conclusions they draw when 
measuring teachers’ MKT. Particular care should be taken when using these measures in other cultural 
settings and more research is needed in this area. We argue that it is important to include the teachers’ 
reflections in order to learn more about their MKT, and more research is needed to investigating teachers’ 
epistemic beliefs (Fives & Buehl, 2008) related to the different aspects of MKT. Analyses of teachers’ 
reflections concerning MKT items can be particularly useful in this regard. Follow-up interviews with 
teachers in groups 1b and 2a would also be relevant for future studies. 


Endnotes 


' The numbers have been changed in our descriptions of the item in order not to reveal the entire item, 
and these details in the teachers’ reflections have been changed accordingly. 

“ECTS stands for European Credit Transfer and Accumulation System. One year of full-time studies 
in Norway gives 60 ECTS. 

> The teachers’ names have been changed to ensure anonymity. 
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